Finish the container header on every rendered export - #473
Merged
Conversation
Builds on Sam's fix. His diagnosis was exact and the approach is his: a container whose muxer finishes the file by seeking back cannot be written to a pipe, so render it to a real file instead. Three things on top. MP3 is affected as well. On a pipe ffmpeg does not write a wrong Xing frame, it writes none at all, and these are VBR (-q:a 2), so the file carries no duration and no seek table. Measured through the real API: a four second export reported 00:00:03.87 before this and 00:00:04.00 after. get_stem_mp3, the MP3 region endpoint sitting next to the WAV one that was corrected, was still on pipe:1 and is now rendered too. FLAC was equally affected and equally unreported: STREAMINFO total samples and all sixteen MD5 bytes were zero, and ffprobe read the duration as N/A. ogg is deliberately left streaming, because a granule position rides on every page and nothing is patched afterwards, and mp4 already muxes fragmented for this exact reason. _render_ffmpeg became _render_to_file, returning a path rather than yielding chunks, so the endpoints answer with FileResponse. That is worth more than the buffering costs. A streamed render commits HTTP 200 before ffmpeg has exited, so a failure reached the client as a truncated file that only the log recorded; the exit code is now known while the response is still ours to choose, and a failed render is an honest 500. That is the limitation #280 documented and could not fix. Content-Length and range requests come along with it. The render cache had to be invalidated. Its key had no version, every entry written before this holds an unpatched header, and _prune_mixdown_cache evicts by age rather than validity, so anyone who exported before upgrading would have been handed the same broken file back for the same parameters indefinitely. _RENDER_CACHE_VERSION is folded into the key so those entries can never be reached again. Verified: 836 passed against 825 on main in a clean worktree, the same four pre-existing local failures on both. 46 browser tests. Each new test was confirmed to fail without the change it covers, by putting MP3 back on the streaming path and watching the Xing assertions break.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #458
Reported and diagnosed by @sam1am, whose commit is the first one here, pulled from his fork because the repo restricts pull requests to collaborators.
His analysis was exact. A muxer that finishes the file by seeking back to a header it wrote earlier cannot do that on
pipe:1, so every WAV we exported kept the0xFFFFFFFFplaceholder and claimed roughly 4 GB of audio. The fix is his: render to a real file instead.It reaches more than WAV
Measured through the real API, exporting the same four second mix in each format and running
ffprobeon the result:Ignoring maximum wav data size, file may be invalidDuration: N/A— STREAMINFO total samples and all 16 MD5 bytes zeroDuration: 00:00:03.87— wrong, andEstimating duration from bitrateMP3 is the one worth arguing about, so here is the argument. On a pipe ffmpeg does not write a wrong Xing frame, it writes none, and these are VBR (
-q:a 2), so there is no duration and no seek table. The 3.87 above is what a player actually reports for a 4.00 second file.get_stem_mp3, the MP3 region endpoint sitting directly beside the WAV one that was corrected, was still onpipe:1and is now rendered too.OGG stays streaming on purpose: a granule position rides on every page and nothing is patched afterwards. MP4 already muxes
frag_keyframe+empty_moovfor exactly this reason and is untouched.Answering with a file, not a generator
_render_ffmpegbecame_render_to_fileand returns a path, so the endpoints reply withFileResponse.The reason is not tidiness. A streamed render commits HTTP 200 before ffmpeg has exited, so a failure reached the client as a truncated file and only the server log knew. Awaiting the render first means the exit code is known while the response is still ours to choose, and a failed export is now an honest 500. That is the limitation #280 documented and could not fix from where it stood.
Content-Lengthand range requests come along with it, which chunked encoding cannot offer at all.The cache would have kept serving the broken files
_mixdown_cache_keyhad no version component. Every entry written before this holds an unpatched header, the key that produced it is still reachable, and_prune_mixdown_cacheevicts by age rather than validity. Without something, a user who exported before upgrading would be handed the same broken file back for the same parameters indefinitely._RENDER_CACHE_VERSIONis folded into the key, so those entries can never be reached again and age out through the existing budget.Verified
mainrun in a clean worktree with the same environment. The same four pre-existing local failures on both, none of them in stems or export.ruff checkandruff format --checkclean.ffprobe, which is the table above.Each new test was confirmed to fail without the change it covers, not merely to pass with it. Putting MP3 back on the streaming path breaks the Xing assertions and the
Content-Lengthone; restoringget_stem_mp3topipe:1breaks the region test on its own.One note for whoever runs these locally: 23 of the tests in
test_stems_api.pyskip silently without a resolvable ffmpeg, and they are the ones that check headers.STEMDECK_FFMPEG_DIRhas to be set or the whole area looks green while testing nothing.